Learning Negotiating Behavior Between Cars in Intersections using Deep Q-Learning

This paper concerns automated vehicles negotiating with other vehicles, typically human driven, in crossings with the goal to find a decision algorithm by learning typical behaviors of other vehicles. The vehicle observes distance and speed of vehicles on the intersecting road and use a policy that adapts its speed along its pre-defined trajectory to pass the crossing efficiently. Deep Q-learning is used on simulated traffic with different predefined driver behaviors and intentions. The results show a policy that is able to cross the intersection avoiding collision with other vehicles 98% of the time, while at the same time not being too passive. Moreover, inferring information over time is important to distinguish between different intentions and is shown by comparing the collision rate between a Deep Recurrent Q-Network at 0.85% and a Deep Q-learning at 1.75%.Comment: 6 pages, 7 figures, Accepted to IEEE International Conference on Intelligent Transportation Systems (ITSC) 201

Reinforcement Learning with Uncertainty Estimation for Tactical Decision-Making in Intersections

This paper investigates how a Bayesian reinforcement learning method can be used to create a tactical decision-making agent for autonomous driving in an intersection scenario, where the agent can estimate the confidence of its recommended actions. An ensemble of neural networks, with additional randomized prior functions (RPF), are trained by using a bootstrapped experience replay memory. The coefficient of variation in the estimated $Q$-values of the ensemble members is used to approximate the uncertainty, and a criterion that determines if the agent is sufficiently confident to make a particular decision is introduced. The performance of the ensemble RPF method is evaluated in an intersection scenario, and compared to a standard Deep Q-Network method. It is shown that the trained ensemble RPF agent can detect cases with high uncertainty, both in situations that are far from the training distribution, and in situations that seldom occur within the training distribution. In this study, the uncertainty information is used to choose safe actions in unknown situations, which removes all collisions from within the training distribution, and most collisions outside of the distribution

Reinforcement Learning in the Wild with Maximum Likelihood-based Model Transfer

In this paper, we study the problem of transferring the available Markov Decision Process (MDP) models to learn and plan efficiently in an unknown but similar MDP. We refer to it as \textit{Model Transfer Reinforcement Learning (MTRL)} problem. First, we formulate MTRL for discrete MDPs and Linear Quadratic Regulators (LQRs) with continuous state actions. Then, we propose a generic two-stage algorithm, MLEMTRL, to address the MTRL problem in discrete and continuous settings. In the first stage, MLEMTRL uses a \textit{constrained Maximum Likelihood Estimation (MLE)}-based approach to estimate the target MDP model using a set of known MDP models. In the second stage, using the estimated target MDP model, MLEMTRL deploys a model-based planning algorithm appropriate for the MDP class. Theoretically, we prove worst-case regret bounds for MLEMTRL both in realisable and non-realisable settings. We empirically demonstrate that MLEMTRL allows faster learning in new MDPs than learning from scratch and achieves near-optimal performance depending on the similarity of the available MDPs and the target MDP

Learning Negotiating Behavior Between Cars in Intersections using Deep Q-Learning

This paper concerns automated vehicles negotiating with other vehicles, typically human driven, in crossings with the goal to find a decision algorithm by learning typical behaviors of other vehicles. The vehicle observes distance and speed of vehicles on the intersecting road and use a policy that adapts its speed along its pre-defined trajectory to pass the crossing efficiently. Deep Q-learning is used on simulated traffic with different predefined driver behaviors and intentions. The results show a policy that is able to cross the intersection avoiding collision with other vehicles 98% of the time, while at the same time not being too passive. Moreover, inferring information over time is important to distinguish between different intentions and is shown by comparing the collision rate between a Deep Recurrent Q-Network at 0.85% and a Deep Q-learning at 1.75%

Development of Components to Realize Control of an Electrostatic Actuator

SammanfattningI Higuchi och Yamamoto laboratoriet på Tokyo Universitetet har det utvecklats en elektrostatiskt ställdon som kan förflytta pappersliknande halvledande material med hjälp av elektrostatiska fält.I denna rapport presenteras utvecklingen av ett antal grundlägande komponenter för att realisera reglering av ett elektrostatiskt ställdon. Genom att använda bildbehandlig samt en kamera kan man fastställa positionen av objektet som styrs av det elektrostatiska ställdonet.Två alternativa bildbehandlings algoritmer studeras, en som använder sig av OpenCV för att filtrera ut en specifik färg och en som använder sig av ARtoolkit för att hitta ett specifikt mönster. ARtoolkit metoden valdes på grund av att den är mindre kännslig till störningar och stabilare än OpenCV metoden.Ett grafiskt gränsnitt samt en PIC mikrokontroller används för att styra det elektrostatiska ställdonet från en dator, med hjälp av USB kommunikation. Den elektrostatiska ställdonet förflyttar objeket med en tophastighet på 86 mm/s. Kombinationen av posisionerings systemet och styrelektroniken möjligör reglering av det elektrostatiska ställdonet.AbstractAt the University of Tokyo, Higuchi and Yamamoto lab, there is an electrostatic actuator that is capable of moving sheet-like semi-conductive material by electrostatic force.This thesis presents the development of a set of basic components to realize control of the electrostatic actuator. The position of the object that is moved by the actuator is tracked by using a camera and image processing algorithms.Two approaches are presented, one by using OpenCV to track a specific color and one by using ARtoolkit to track a specific marker. The ARtoolkit method is chosen because it is less sensitive to noise and is more stable than the OpenCV method.A PIC micro-controller with an interface on a PC is implemented to allow a computer program to control the electrostatic actuator. Using only the PIC controller and a computer program, the actuator could operate at about 86 mm/s as its maximum speed. The combination of the PIC controller and the position detection program module will allow various motions of the electrostatic motor

Development of Components to Realize Control of an Electrostatic Actuator

SammanfattningI Higuchi och Yamamoto laboratoriet på Tokyo Universitetet har det utvecklats en elektrostatiskt ställdon som kan förflytta pappersliknande halvledande material med hjälp av elektrostatiska fält.I denna rapport presenteras utvecklingen av ett antal grundlägande komponenter för att realisera reglering av ett elektrostatiskt ställdon. Genom att använda bildbehandlig samt en kamera kan man fastställa positionen av objektet som styrs av det elektrostatiska ställdonet.Två alternativa bildbehandlings algoritmer studeras, en som använder sig av OpenCV för att filtrera ut en specifik färg och en som använder sig av ARtoolkit för att hitta ett specifikt mönster. ARtoolkit metoden valdes på grund av att den är mindre kännslig till störningar och stabilare än OpenCV metoden.Ett grafiskt gränsnitt samt en PIC mikrokontroller används för att styra det elektrostatiska ställdonet från en dator, med hjälp av USB kommunikation. Den elektrostatiska ställdonet förflyttar objeket med en tophastighet på 86 mm/s. Kombinationen av posisionerings systemet och styrelektroniken möjligör reglering av det elektrostatiska ställdonet.AbstractAt the University of Tokyo, Higuchi and Yamamoto lab, there is an electrostatic actuator that is capable of moving sheet-like semi-conductive material by electrostatic force.This thesis presents the development of a set of basic components to realize control of the electrostatic actuator. The position of the object that is moved by the actuator is tracked by using a camera and image processing algorithms.Two approaches are presented, one by using OpenCV to track a specific color and one by using ARtoolkit to track a specific marker. The ARtoolkit method is chosen because it is less sensitive to noise and is more stable than the OpenCV method.A PIC micro-controller with an interface on a PC is implemented to allow a computer program to control the electrostatic actuator. Using only the PIC controller and a computer program, the actuator could operate at about 86 mm/s as its maximum speed. The combination of the PIC controller and the position detection program module will allow various motions of the electrostatic motor

Reinforcement Learning with Uncertainty Estimation for Tactical Decision-Making in Intersections

This paper investigates how a Bayesian reinforcement learning method can be used to create a tactical decision-making agent for autonomous driving in an intersection scenario, where the agent can estimate the confidence of its decisions. An ensemble of neural networks, with additional randomized prior functions (RPF), are trained by using a bootstrapped experience replay memory. The coefficient of variation in the estimated Q-values of the ensemble members is used to approximate the uncertainty, and a criterion that determines if the agent is sufficiently confident to make a particular decision is introduced. The performance of the ensemble RPF method is evaluated in an intersection scenario and compared to a standard Deep Q-Network method, which does not estimate the uncertainty. It is shown that the trained ensemble RPF agent can detect cases with high uncertainty, both in situations that are far from the training distribution, and in situations that seldom occur within the training distribution. This work demonstrates one possible application of such a confidence estimate, by using this information to choose safe actions in unknown situations, which removes all collisions from within the training distribution, and most collisions outside of the distribution

Learning When to Drive in Intersections by Combining Reinforcement Learning and Model Predictive Control

In this paper, we propose a decision making algorithm intended for automated vehicles that negotiate with other possibly non-automated vehicles in intersections. The decision algorithm is separated into two parts: a high-level decision module based on reinforcement learning, and a low-level planning module based on model predictive control. Traffic is simulated with numerous predefined driver behaviors and intentions, and the performance of the proposed decision algorithm was evaluated against another controller. The results show that the proposed decision algorithm yields shorter training episodes and an increased performance in success rate compared to the other controller

Reinforcement Learning in the Wild with Maximum Likelihood-based Model Transfer

In this paper, we study the problem of transferring the available Markov Decision Process (MDP) models to learn and plan efficiently in an unknown but similar MDP. We refer to it as \textit{Model Transfer Reinforcement Learning (MTRL)} problem. First, we formulate MTRL for discrete MDPs and Linear Quadratic Regulators (LQRs) with continuous state actions. Then, we propose a generic two-stage algorithm, MLEMTRL, to address the MTRL problem in discrete and continuous settings. In the first stage, MLEMTRL uses a \textit{constrained Maximum Likelihood Estimation (MLE)}-based approach to estimate the target MDP model using a set of known MDP models. In the second stage, using the estimated target MDP model, MLEMTRL deploys a model-based planning algorithm appropriate for the MDP class. Theoretically, we prove worst-case regret bounds for MLEMTRL both in realisable and non-realisable settings. We empirically demonstrate that MLEMTRL allows faster learning in new MDPs than learning from scratch and achieves near-optimal performance depending on the similarity of the available MDPs and the target MDP